---
title: "HCBS Quality: Person-Reported Outcomes"
output: 
  flexdashboard::flex_dashboard:
    storyboard: true
    theme: yeti
    source: embed
---

```{r setup, include=FALSE}
library(tidyverse); library(flexdashboard); library(plotly)

ref_measure_groups <- 
  read_csv("../data/mapping_questions.csv") %>%
  mutate(
    field_desc = str_trim(field_desc),
    field_desc = str_replace(field_desc," who do not have disabilities\\?$","?"),
    sub_domain = recode(
      sub_domain,
      `Choice of provider` = "Choice of provider and services",
      `Choice of services` = "Choice of provider and services"
    )
  ) %>%
  # Exclude questions removed from future surveys (to allow comparison over time)
  filter(
    !participant_field %in% c(
      "Q40","Q79","Q62","Q64","Q66","Q70","Q50","Q53",
      "Q112","Q22","Q89","Q90","Q93","Q100"
    )
  ) %>%
  rename(
    field = participant_field,
    measure_group  = domain,
    measure_domain = sub_domain
  )

df <- list()

df$df_all <- 
  feather::read_feather("../data/hcbs_long.feather") %>%
  # Join to subset of domain fields
  inner_join(
    ref_measure_groups %>% select(field,measure_group,measure_domain), 
    by = "field"
  ) %>%
  select(-wsa_id) %>%
  # Exclude provider responses not mapped to a participant response
  filter(is.na(response_id) == F) 

```

```{r make_base_df}

# Create df of all eligible responses for use in measures, one row per response ID/field

df$df_field <-
  df$df_all %>%
  group_by(measure_group,measure_domain,field,field_desc,response) %>%
  mutate(n = n_distinct(response_id)) %>%
  group_by(measure_domain,field,field_desc) %>% 
  filter(!is.na(measure_domain)) %>%
  # Remove missing responses and "Not applicable" responses
  filter(!is.na(response) & !response %in% c("Do not need","Does not apply")) %>%
  mutate(
    pct = round(n / sum(n) * 100, digits = 1),
    ambiguous = !response %in% c("Yes","No"),
    # Calculate % of responses to the question which are ambiguous 
    pct_ambiguous = sum(pct[ambiguous]),
    # Calculate n of responses to the question which are clear
    n_clear = sum(n[!ambiguous])
  ) %>%
  ungroup() %>%
  # Remove questions with a high pct of ambiguous responses or a small n of clear responses
  filter(pct_ambiguous <= 25 & n_clear >= 100) %>%
  # For the remaining questions, include only "Yes"/"No" answers for calculation
  filter(response %in% c("Yes","No")) %>%  
  select(response_id,measure_group,measure_domain,field,field_desc,response) %>%
  mutate(response_log = response == "Yes") %>%
  group_by(response_id,field_desc,measure_domain,measure_group) %>%
  summarize(
    valid_responses = n(),
    pos_responses = sum(response_log)
  )

# Create df summarizing responses at the group level and 
# determining consistency of responses within each level (i.e. 'boxes')

df$df_composite <-
  df$df_field %>%
  group_by(response_id,measure_domain,measure_group) %>%
  summarize(
    valid_responses = sum(valid_responses),
    pos_responses = sum(pos_responses)
  ) %>%
  mutate(
    top_box = pos_responses == valid_responses,
    bottom_box = pos_responses == 0,
    middle_box = top_box == F & bottom_box == F
  )

# Create summary df of excluded participant responses, by response/field
df$df_excluded <-
  df$df_all %>%
  group_by(measure_group,measure_domain,field,field_desc,response) %>%
  summarize(n = n_distinct(response_id)) %>%
  group_by(measure_domain,field,field_desc) %>%
  mutate(pct = round(n / sum(n) * 100, digits = 1)) %>%
  filter(!response %in% c("Yes","No")) %>%  
  filter(is.na(measure_domain) == F)

```

```{r make_measure_df}

measure_df <-
  df$df_composite %>%
  group_by(measure_group,measure_domain) %>%
  summarize(
    total_answers     = sum(valid_responses),
    positive_answers  = sum(pos_responses),
    total_surveys     = n_distinct(response_id),
    topbox_surveys    = sum(top_box),
    bottombox_surveys = sum(bottom_box),
    middlebox_surveys = sum(middle_box)
  ) %>%
  mutate(
    weighted_pct    = round(positive_answers / total_answers * 100, digits = 1),
    topbox_score    = round(topbox_surveys / total_surveys * 100, digits = 1),
    bottombox_score = round(bottombox_surveys / total_surveys * 100, digits = 1),
    middlebox_score = round(middlebox_surveys / total_surveys * 100, digits = 1)
  ) %>%
  inner_join(
    ref_measure_groups %>% group_by(measure_domain) %>% summarize(questions = n_distinct(field)), 
    by = "measure_domain"
  )

group_df <-
  measure_df %>%
  group_by(measure_group) %>%
  summarise_at(vars(total_answers,positive_answers),list(~sum(.))) %>%
  mutate(weighted_pct    = round(positive_answers / total_answers * 100, digits = 1))

question_df <-
  df$df_all %>%
  filter(response %in% c("Yes","No")) %>%
  mutate(
    response = fct_relevel(response,"Yes","No"),
    field_desc = str_trunc(field_desc,50,"right")
  ) %>%
  # Recalculate % with exclusions removed
  group_by(measure_group,measure_domain,field_desc,response) %>%
  summarize(n = n_distinct(response_id)) %>%
  spread(response,n, fill = 0) %>%
  mutate(pct = round(Yes / (Yes + No) * 100, digits = 1)) %>%
  ungroup() %>%
  mutate_at(
    vars(measure_group,measure_domain,field_desc),
    list(~as.factor(.))
  ) 

```

### Guided by values
 Values informing what we mean by quality

The HCBS quality workgroup is a subgroup of the HCBS *Implementation Advisory Group (IAG)*, a diverse group of stakeholders affected by or invested in the implementation of the Home and Community Based Services (HCBS) rule. Members of this group volunteered to provide input to the department to guide the development and use of an initial set of quality measures. 

The following values are central to the highest quality of life and positive outcomes for people served by Home and Community Based Services:

**Person FIRST**:

‘People first’ language emphasizes that people with disabilities are – first and foremost – people. People are not defined by their diagnosis, and have individual abilities, interests, needs and life goals. This also includes that:

- All people are treated with dignity and respect. Individual rights are promoted and protected as people seek to achieve their personal, life dreams and goals.
- Cultural sensitivity and competency honors diversity of all people and assures equal access to supports and services.
- People are supported in their physical, mental, spiritual/social well-being including building healthy, meaningful relationships. They are connected to others.

**Person CENTERED**:

- Person-centered planning is strength-based and focuses on people achieving their fullest potential. This means that people are involved in the planning, implementation, and evaluation of their services and supports.
- Persons served are at the center of a circle of supports, which includes supporting them in establishing long-term relationships that form the foundation for much of our social, personal and spiritual lives. Trusted relationships are important so that people can communicate concerns about their physical, mental and social wellbeing.

**Person DRIVEN**:

Supports and services promote independence, self-determination, and freedom where people are empowered with individual choice, control and decision-making in building their own meaningful life in their community. 

- *Home*: People are supported in choosing a ‘place of one’s own’ in their community, where both the home and whomever lives there, or provides support, are chosen by the person. 
- *Community*: People are supported in full community participation and belonging, where people are involved in local activities and organizations, including having meaningful, integrated employment and educational opportunities. Community participation is tied to mental and social wellbeing, self-esteem, relationship building, and being an important part of one’s community.


### Quality of life in community settings
 High-level overview from HCBS Surveys {data-commentary-width=600}

This document is intended to provide a sample version of how survey responses might be used to inform statewide understanding of quality in HCBS settings.  The information presented here is derived from the Habilitation Supports Waiver (HSW) surveys, whose development and fieding are [described here](https://ddi.wayne.edu/hcbssurvey).  Let's start with what a high-level summary might look like...

The chart below shows survey responses from HCBS participants which have been summarized into broad groups related to quality of life areas.  As this is a summary of many individual responses, it is important to understand which questions make up these areas, and how people's responses to these questions are combined.  

You can scroll through the remaining pages (*see boxes above*) to see more details about how these measures are composed, and look to the panel on the right to explore which questions are grouped together.

```{r}

group_df %>%
  plot_ly(
    x = ~weighted_pct,
    y = ~fct_reorder(measure_group,weighted_pct),
    color = I("#3B9AB2"),
    orientation = 'h',
    width = 800, height = 400
  ) %>%
  add_bars(
    hoverinfo = 'text',
    text = ~paste(
      "Out of a total of ","",total_answers," responses
",
      "to questions related to ", tolower(measure_group),",
",
      "",positive_answers," (",weighted_pct,"%)"," indicated a positive experience"
    )
  ) %>%
  layout(
    xaxis = list(
      title = "Weighted composite %",
      range = c(0,100)
    ),
    yaxis = list(
      title = "",
      tickfont = list(size = 12)
    ),
    margin = list(
      l = 250, r = 50,
      b = 60,  t = 50, pad = 4
    )
  )

```

***

Questions in the HCBS survey were mapped to existing *quality of life* frameworks in order to assure their broader relevance and to allow for consistent communication.  This process is outlined in greater detail in the [HCBS Quality Measures](https://tbdsolutions.github.io/misc_presentations/hcbs_quality_measures.html) document.

Interact with the diagram below to see which composite measures and questions fall within each grouping, or go to the right-most box above to get *all* of the grouping details:

```{r}
df$df_field %>%
  group_by(measure_domain,measure_group,field_desc) %>%
  summarize(n = n_distinct(response_id)) %>%
  mutate(field_desc = str_trunc(field_desc,40,"right")) %>%
  collapsibleTree::collapsibleTree(
    hierarchy = c("measure_group","measure_domain","field_desc"),
    root = "Life Areas",
    nodeSize = "leafCount",
    width = 900,
    zoomable = T
  )
```


### Combining similar questions
 Sample measures using personal survey feedback {data-commentary-width=500}

The barchart below shows people's responses on survey questions which have been grouped into composite measures (*see description on the right*).  The bars are colored based on the quality of life domain which they belong to, as shown on the preceding page.

```{r}
measure_df %>%
  group_by(measure_group) %>%
  plot_ly(
    x = ~weighted_pct,
    y = ~fct_reorder(measure_domain,weighted_pct),
    color = ~measure_group,
    colors = paletteer::paletteer_d("ggthemes",`Tableau 20`),
    orientation = 'h',
    width = 900, height = 500
  ) %>%
  add_bars(
    hoverinfo = 'text',
    text = ~paste(
      "Out of a total of ","",total_answers," responses
",
      "to questions related to ", tolower(measure_group),",
",
      "",positive_answers," (",weighted_pct,"%)"," indicated a positive experience"
    )
  ) %>%
  layout(
    xaxis = list(
      title = "Weighted composite %",
      range = c(0,100)
    ),
    yaxis = list(
      title = "",
      tickfont = list(size = 12)
    ),
    legend = list(
      # orientation = 'h',
      # x = 0.5, y = -0.15
      x = 1.1, y = 0.5
    ),
    margin = list(
      l = 250, r = 50,
      b = 60,  t = 50, pad = 4
    )
  )
```

*** 

**What is a composite measure?**

There are lots of reports that we encounter on a day-to-day basis, and each one makes new demands on our attention.  In order to decrease the total number of measures requiring interpretation by stakeholders, a measure can combine similar survey questions into groups which are similar and conceptually coherent.  Measures made by combining questions in this way are called *composite measures*.

**Different ways of summarizing people's feedback**

There are two different summary scores calculated for each composite measure:

- *Composite Score*:  The composite score proportion of eligible responses which indicated a positive experience.  For instance, if a person answered four questions about their "Freedom to come and go", and answered that they had such freedoms in 3 out of 4 of their answers, then the composite score for that person related to that group of questions would be 3/4, or 75%.
- *Consistent Performance Indicator*: On this page and the previous page, we saw a visualization of composite measures.  The next page will show the *Consistent Performance Indicator* approach.

**Considerations**
It is important to understand the following details when interpreting this measure:

- *Based on participant responses*: These measures are based on responses from people who receive services, not from the providers who answered similar survey questions.
- *Approach to weighting responses*: There are two basic options to weighting composite survey responses: equal or unequal.  Equal weighting treats all questions in the composite as equally important, even though some items may be answered more frequently than others. Unequal weighting suitably discounts items with a lower volume of responses, yielding a composite that is more statistically precise.  Since response rates to the HCBS survey questions are different across questions, we use an unequal rating strategy.

### Consistency
 Do people have a consistently positive experience in their homes and workplaces?  {data-commentary-width=500}

Many of us know from our own lives that we might have positive experiences in one area but not in others, and that even isolated negative experiences can impact our quality of life.

The graph below shows the consistency of personal experience within each of the composite groups:

```{r consistent}

measure_df %>%
  ungroup() %>%
  select(
    measure_domain,
    starts_with("topbox"),starts_with("middlebox"),starts_with("bottombox"),
    total_surveys,weighted_pct
  ) %>%
  gather(
    score_type,score,-measure_domain,-weighted_pct,-ends_with("surveys")
  ) %>%
  mutate(
    score_type = recode(
      score_type,
      `topbox_score` = "consistently positive",
      `middlebox_score` = "mixed",
      `bottombox_score` = "consistently negative"
    ),
    score_type = fct_relevel(
      score_type,
      "consistently positive","mixed","consistently negative"
    )
  ) %>%
  plot_ly(
    x = ~score,
    y = ~fct_reorder2(
      measure_domain,
      .x = as.numeric(score_type),
      .y = score
    ),
    color = ~score_type,
    colors = c("#3B9AB2", "#78B7C5", "#EBCC2A", "#E1AF00", "#F21A00"),
    orientation = 'h',
    width = 800, height = 500
  ) %>%
  add_bars(
    hoverinfo = 'text',
    text = ~paste0(
      score,"% of respondents had
",
      score_type," experiences
",
      "related to ", tolower(measure_domain)
    )
  ) %>%
  layout(
    showlegend = FALSE,
    xaxis = list(title = "% of survey respondents"),
    yaxis = list(
      title = "",
      tickfont = list(size = 12)
    ),
    autosize = F, 
    margin = list(
      l = 150, r = 50,
      b = 90,  t = 40, pad = 4
    ),
    barmode = 'stack'
  )

```

***

**Are my experiences of my home and workplace consistently positive?**

If our goal is that people's experience of HCBS settings be consistently positive in relationship to the areas outlined by the HCBS Final Rule, then it is important to look at an individual's experience across all of the questions that address a particular topic.  

These *Consistent Performance Indicator* scores look at how often people's experience was *consistently positive* (i.e. they responded 'Yes' to each question that made up the measure), or *consistently negative* (i.e. they responded 'No' to all of the questions that made up the measure).  Individuals may also have had mixed experiences, meaning that their responses to the questions in a given group included both positive and negative responses. 

**Considerations**

A few considerations related to this method of summarizing experience are as follows:

- Since this method sets a higher expectation for people's homes and workplaces, the scores for consistent positive performance will tend to be lower.
- Composite groups with only a single question will have no surveys which fall into the "mixed experience" group.
- Since the number of possible positive responses is limited to the number of questions answered, people who answer fewer questions will be more likely to fall into the top box (*if they answer positively on a few questions*) or bottom box (*if they answer negatively on a few questions*).

### What's in a measure?
 Breakdown of specific survey questions in each measure {data-commentary-width=400}

This panel shows which specific questions make up the sample measures, and how survey respondents answered each of these questions.  As mentioned in previous pages, composite measures are made up of various survey questions.  The table below shows the following information for each survey question:

- *QoL Domain*: Which quality of life area is it grouped in?
- *Composite Measure*: Which composite measure is it a part of?
- *Yes*: Number of participant responses of "Yes" to this question.
- *No*: Number of participant responses of "No" to this question.
- *% Positive*: Proportion of total "Yes" or "No" responses which were answered with a "Yes".

```{r question_barchart, echo=FALSE}

question_df %>%
  DT::datatable(
    rownames = FALSE,
    filter = "top",
    colnames = c(
      'QoL Domain', 
      'Composite Measure', 
      'Question',
      'Yes',
      'No',
      '% positive'
    ),
    extensions = c('Responsive')
  ) %>%
  DT::formatStyle(
    'pct',
    background = DT::styleColorBar(question_df$pct, 'steelblue'),
    backgroundSize = '100% 90%',
    backgroundRepeat = 'no-repeat',
    backgroundPosition = 'center'
  )

```

*** 

**Exclusions**

It is important to understand the following details when interpreting these measures, since not all participant responses are included in the calculation of the measures.  Specifically,

- *Missing responses are excluded*:  Since survey responses were voluntary for participants, not all respondents provided answers to every question.  Because of this, the number of total responses can vary by question.
- *Some other response options are filtered out*: Most questions on the survey ask for *Yes* or *No* responses.  For some questions, however, there are additional options such as *I don't know* or *Does not apply*.  
- *Questions with a high rate of ambiguous responses are excluded altogether*: Questions are excluded from use in the quality measures if >=25% of relevant responses (i.e. responses which are not missing or marked as 'does not apply' by the participant) are ambiguous, defined here as an answer other than "Yes" or "No".  In addition, the number of unambiguous responses to a question must be greater than 100 in order to keep a single response from having an outsized impact on the calculation of a particular measure. 

A table summarizing excluded responses is shown below:

```{r excluded}
df$df_excluded %>% 
  ungroup() %>%
  filter(!response %in% c("Yes","No")) %>%
  select(field_desc,response,n,pct) %>%
  mutate(response = ifelse(is.na(response),"Missing",response)) %>%
  group_by(response) %>%
  summarize(n = sum(n)) %>%
  ungroup() %>%
  mutate(pct = round(n / sum(n) * 100, digits = 1)) %>%
  arrange(desc(pct)) %>%
  knitr::kable(
    col.names = c("Response","n","%")
  )
```

### What other questions are on the survey?
 Reference of available survey questions and domains {data-commentary-width=400}

```{r ref_tbl}
library(DT)

ref_measure_groups %>%
  select(measure_group,measure_domain,field_desc,field) %>%
  mutate_at(vars(measure_group,measure_domain),list(~as.factor(.))) %>%
  group_by_all() %>%
  DT::datatable(
    rownames = FALSE,
    filter = "top",
    colnames = c(
      'QoL Domain', 
      'Composite Measure', 
      'Question Text',
      'Question ID'
    ),
    extensions = c('Responsive')
  )

```

*** 

**Other Questions on the Survey**

The sample measure that was discussed in the previous panels uses only a few of the many questions that are available on the HCBS Participant Survey.  On the table to the right, you can see each of the questions from that survey choice has been mapped to a Quality of Life (QoL) domain, and the specific clusters of questions which have been grouped together for consideration as potential *Composite Measure*s.  If you filter the *Composite Measure* and select a single option, the table will display all of the questions which were flagged as related to that characteristic of quality.

**Person-Reported Outcomes**

In quality measurement, participant surveys such as the MDHHS and MI-DDI HCBS Survey are referred to as [*person-reported outcomes*](https://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx) or [*experience of care*](http://blog.ncqa.org/the-q-series-what-are-the-types-of-quality-measures/) measures.  According to [AHRQ](https://www.ahrq.gov/cahps/about-cahps/patient-experience/index.html), these measures are intended to tell us "*whether something that should happen...actually happened or how often it happened*": a question which the person receiving services and supports is uniquely qualified to answer.

### How were the measures developed?
 Documentation and Considerations {data-commentary-width=400}

The Michigan Department of Health and Human Services, Behavioral Health and Developmental Disabilities Administration (MDHHS-BHDDA) requested that TBD Solutions develop a pragmatic approach for measuring quality related to HCBS service settings affected by the [HCBS Final Rule](https://www.federalregister.gov/d/2014-00487).  While the initial focus of MDHHS-BHDDA has been on assuring compliance, their longer-term strategy for use of HCBS data is oriented to improve quality of life through a focus on participants' personal experience of settings and services. 

The intent of this pragmatic approach was to work with existing datasets in order to develop usable measures, rather than to propose the collection of new data.  This approach, while precluding the development or fielding of new surveys or tools, has the benefit of maximizing the use of existing resources, reducing response burden for program participants and staff, and allowing for historical comparisons without requiring substantial delays for data collection and processing.

This initial prototype of person-reported outcomes measures was developed as part of a collaborative, multi-session effort with HCBS quality workgroup, a subgroup of the HCBS Implementation Advisory Group (IAG).  Please note that the measures outlined here are a subset of a broader approach which encompasses various quality of life domains and includes structural measures to identify the impacts of HCBS Final Rule implementation. They fit into the group of measures referred to in that document as [Person-Reported Outcomes](https://tbdsolutions.github.io/misc_presentations/hcbs_quality_measures.html#types_of_quality_measures_proposed), and are intended to serve as a first step toward a comprehensive strategy to measure the quality of HCBS in Michigan's public behavioral health system.

A detailed description of the methodology for conceptualizing and prioritizing measures for development can be found in the [documentation here](https://tbdsolutions.github.io/misc_presentations/hcbs_quality_measures.html).

This prototype was developed by: [![](tbdSolutions-logo.png)](www.tbdsolutions.com)

... in collaboration with: [![](mdhhs_logo.jpg)](http://www.michigan.gov/mdhhs/0,5885,7-339-71550_2941-146590--,00.html)